In general, augmented reality is a technology for showing computer generated graphics (virtual images) overlapped with a real image to make the viewer feel as if the computer generated graphics is in the real world. The AR technology aims to complement the real world with the virtual realm, and the majority of the augmented reality information resides in the real image that the human eyes see.
The computer-generated virtual image may contain a variety of graphical or non-imagery information such as text in registration with an object image besides the ordinary real image. Such computer-generated virtual image may be a marker image of an object with markers attached to include the object's positional information for aligning the virtual image with the real image, a stereo image containing three-dimensional geometric information of the scene to subsequently extract the coordinate data, or existing medical images such as CT, MRI, and PET. Other than the obtained images before the AR image registration, a special image (inclusive of a processed real image) generated in real time while capturing a real image may be used as a virtual image (computer image) for realizing the AR. The special image may be a fluorescent image which receives extensive attention from several fields. In particular, a near infrared (NIR) fluorescent image is widely used in the biomedical field. This wealth of information can be well applied simultaneously to construct a computer-generated virtual image.
The method to overlap virtual images onto the real image that the human eyes see is implemented by the usage of a semi-translucent display, augmented reality glasses, or a head-mount display (HMD).
An exact registration or alignment of the virtual image with the real image is required for realizing augmented reality. It is required to have the virtual image that spatially matches in alignment with the real image exactly to realize the augmented reality. Further, it is necessary to have the geometric information to responsibly align the virtual image onto the real image in the same space coordinate in real-time with the change in the distance or the orientation of the subject and the projector. In case of an HMD, three-dimensional computer graphics data is needed to provide an aligned virtual image projection onto the real image regardless of the distance or orientation with respect to the subject. In case of a semi-translucent display or augmented reality glasses, to render a subject behind the semi-translucent display or augmented reality glasses in registration with computer graphics on the relevant display, a computer-generated virtual image needs to be projected taking account of the observer's line of sight and the distance from the subject, which necessitates three-dimensional computer-generated virtual image information in spatial registration with the subject behind.
Providing viewing experience which is natural and accurate without discomfort or distortion of reality matters much in the augmented reality composed of the computer-generated virtual image and the real image. This applies especially to the AR system used in precise medical operations or hazardous works. For example, the AR system is not supposed to interfere with the delicate medical operation. However, an HMD or a semi-translucent screen is in between the operation site and the operator's eyes so it is prone to be contaminated and may cause a sense of visual discord by interfering with the line of sight partially.
The augmented reality system of some embodiments of the present disclosure directly projects and aligns the computer-generated virtual image onto the subject, unlike the method which uses a semi-translucent display for the virtual image projection including the registration of the projected virtual image with the image of the subject behind the semi-translucent display. Generating an AR image by casting (projecting) a virtual image directly on the subject may realize an augmented reality with less sense of discomfort in performing high-accuracy or high-risk operation because the observer is to see the actual subject.
A conventional AR system as depicted in FIGS. 4 and 5 needs to be comprised of a device to generate computer-generated virtual image including data to be added to the real image; a (real-virtual image alignment) device for aligning (calculating and transforming) the coordinates of the computer-generated virtual image to that of the real image; and a projection device of the AR image which contains the real image overlapped with the computer-generated virtual image.
Augmented reality image projection system, which projects an image to the subject, needs a projection device further to a typical AR system. As a real image capture device and the projection device may not be in alignment, a device (alignment device for aligning the computer-generated virtual image to the subject) is additionally necessary for aligning the subject and the computer-generated virtual image.
However, a conventional augmented reality image projection system, which is a side-by-side arrangement of two dedicated units of a projection device and a real image capture device, has following issues.
First, it involves a complicated process of aligning the projection onto the subject. The lack of unity between the projection device for projecting the AR image and the real image capture device which serves as an alignment reference results in their discrepancies in direction and distance. The AR image cannot be projected as it is, and needs an alignment process for registration with the subject. For the alignment registration, a special algorithm may be used to compare and match the captured subject image and the AR image in real time. A method using a sort of markers or stereoscopic cameras may also be used as described in Korean Patent Publication No. 10-0726028 (May 31, 2007). Such markers or stereoscopic cameras provide the positional information of the subject as a tracker. The information provided by the tracker is fed to the system to align the AR image with the subject by adjusting the size, position, or angle of orientation. Such projection system may still have discomfort for the viewer as there may exist a time delay in the projected image. It takes a lot of calculations to get an image for projection if there is a real-time change of distance or direction between the subject and the projection system, or the subject is expected to move relentlessly. In addition, as the subject is three-dimensional having a certain thickness, the distance of the subject from the projection system differs portion by portion, which requires an additional calculation time for alignment even if a stereoscopic camera is in use. A fundamental shortcoming is shown in FIG. 5 that the projection system has blind spots where the projection system cannot reach due to an angular difference between the line of sight and the line of projection.
Second, a size of the system is hard to be reduced in compact form if each device is configured independently. As the separately installed camera and projection device may occupy a large space, 1) it may block the line of sight of the observer who needs to see an AR image projected onto the subject, which limits the usability of the system. Size reduction of the AR image projection system is not feasible in typical projection system having separate devices for projecting and imaging including positional information. Such cases include 2) when an endoscopic or a laparoscopic device is in use to view the subject, or 3) when the system is to be configured in places like an astral lamp. The size of the system can play a decisive role in particular applications.
Third, it is technically difficult to implement additional real-time imaging device such as a fluorescence imaging device. Additional imaging device requires additional trackers or complicated calculation for image alignment. Physical realization is hard as additional devices are necessary according to the additional number of images. For example, devices for capturing a thermal image of the subject and then projecting, or capturing a fluorescent image of the subject and then projecting may be added side by side to a subject image capture device and a projection device. It becomes very complicated system occupying a lot of space.
Fourth, interference between the projection image and the illumination of the subject diminishes the contrast of projected AR image. Additional fluorescence image further makes it difficult to configure the system as it interferes with the excitation light source.
Illumination is always necessary for both the conventional augmented reality image projection system or the augmented reality image projection system of the present disclosure as both need to take a real image for alignment. The real image serves as a basis for alignment in augmented reality image. In general, the wavelength range of illumination for capturing the real image and that of the projected image from the augmented reality image projection system overlap. In this case, increasing the intensity of the illumination to improve the captured image quality 1) makes the projected image less visible and 2) results in an adverse effect on alignment as the projection image is taken together with the real image. Such interference necessitates a sophisticated device for separately controlling the illuminating device to project a good contrast image. Such device includes a complicated method using time-division illumination and toggling between illumination and image projection to prevent such interference. However, this approach makes the system complex and also brings serious inconvenience to the user's view due to the flashing illumination. 3) In case a fluorescent image is appended, an additional time division is required as the white light illumination interferes with the excitation illumination which prevents simultaneous acquisition and projection of the images.
Placing wavelength range of the illumination for the real image in invisible light range (in particular, infrared range) can fundamentally solve the interference issue of the projected computer-generated AR image and the excitation source for the fluorescence without a complicated time-division scheme. Especially for the AR image projection system according to some embodiments of the present disclosure, real image taken by illuminating with invisible light source suits better as its primary purpose is for image alignment. Previous AR projection system is unable to solve the interference issue between illumination, projection image, and special imaging. Such interference is a problem needed to be solved to build a practical working system.